System and method for adaptive generation using feedback from a trained model

ABSTRACT

A system and method for training a system or a model using any combination of synthetic data and real data. The various aspects of the invention include generation of data that is used to supplement or augment real data, wherein the subject of the data can be segmented. Labels or attributes are automatically added to generated data. The generated data is synthetic data that is generated using seed or real data as well as from other synthetic data. Using the synthetic data, various domain adaptation models can be used and trained.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No.62/827,860 titled SYSTEM AND METHOD FOR ADAPTIVE GENERATION USINGFEEDBACK FROM A TRAINED MODEL filed on Apr. 2, 2019, AND is acontinuation of U.S. Non-Provisional application Ser. No. 16/839,059filed on Apr. 2, 2020 and titled SYSTEM AND METHOD FOR VISUALRECOGNITION USING SYNTHETIC TRAINING DATA, the entire disclosures ofwhich are incorporated herein by reference.

FIELD OF THE INVENTION

The invention is in the field of computer systems and, morespecifically, related to using feedback from a model using datasets fortraining.

BACKGROUND

Known systems that for performing specific tasks based on analysis ofimages are typically trained over a period of time. As the system istrained, the system provides feedback. However, adaptively generatingnew training datasets based on the feedback is currently inefficient ornon-existent. Furthermore, know approaches to creating an accurate,diverse, and inclusive new training dataset require a range of imagesthat include the subject in order to represent the full diversity, whichis time consuming and inefficient to generate. Therefore, what is neededis a system and method that allows for adaptive generation of a largevolume of diverse new training dataset based on feedback from a system,which dataset needs to be generated quickly while representing a highstandard of photorealistic diversity.

SUMMARY OF THE INVENTION

The invention discloses a system and method for adaptive generation of alarge volume of diverse new training data based on feedback from asystem. The generated new training data is created in a short period oftime. The various aspects of the invention include generation of robustnew training dataset using synthetic data or images. Synthetic data isartificially generated data that mimics real-world sensory input. As thesynthetic data is generated, labels are used to provide accurateinformation as part of the dataset that is used for training the system.This creates a robust, diverse, and extensive dataset that will allowdevelopment and training of the system in a significantly more costeffective and time sensitive manner.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more fully understand the aspects and embodiments of theinvention, reference is made to the accompanying drawings. Understandingthat these drawings are not to be considered limitations in the scope ofthe invention, the presently described aspects and embodiments and thepresently understood best mode of the invention are described withadditional detail through use of the accompanying drawings, in whichlike numbers represent the same or similar elements.

FIG. 1 shows a process for generating synthetic data in accordance withthe various aspects and embodiments of the invention.

FIG. 2A show a process for analyzing and providing feedback inaccordance with the various aspects and embodiments of the invention.

FIG. 2B shows a process for segmenting real data to generate syntheticdata in in accordance with the various aspects and embodiments of theinvention.

FIG. 3 shows various synthetic data generated for training a system inaccordance with the various aspects and embodiments of the invention.

FIG. 4 shows a system for developing hybrid datasets using generativemodels (e.g., GANs) to refine synthetic data in accordance with thevarious aspects and embodiments of the invention.

FIG. 5 shows a neural network of the system of FIG. 4 in accordance withthe various aspects and embodiments of the invention.

FIG. 6A shows a system using an adaptive process that changes based onthe properties of obtained synthetic data in accordance with variousaspects and embodiments of the invention.

FIG. 6B shows a process for generating feedback on a training dataset inaccordance with the various aspects and embodiments of the invention.

FIG. 7 shows a process for segmenting real images and combining thesegmented images to generate synthetic image in accordance with thevarious aspects and embodiments of the invention.

DETAILED DESCRIPTION

To the extent that the terms “including”, “includes”, “having”, “has”,“with”, or variants thereof are used in either the detailed descriptionor the claims, such terms are intended to be inclusive in a similarmanner to the term “comprising”.

The ranges of values provided herein do not limit the scope of thepresent invention. It is understood that each intervening value, betweenthe upper and lower limit of that range and any other stated orintervening value in that stated range, is encompassed within the scopeof the invention. The upper and lower limits of these smaller ranges mayindependently be included in the smaller ranges and are also encompassedwithin the invention, subject to any specifically excluded limit in thestated range. Where the stated range includes one or both of the limits,ranges excluding either or both of those included limits are alsoincluded in the invention.

In accordance with the various aspects and embodiments of the invention,“seed data” and “captured data” are used in relation to data that isreal data. The real data may come from any source, including video, realdynamic images and real static images. In accordance with one aspect ofthe invention, real data includes real objects in any setting orenvironment, including native or natural environment or unnaturalenvironment. For example, a cow is naturally found outdoors in a fieldor pasture. The same cow may also be standing in a living room next to atable, which is an unnatural environment for the cow. In accordance withthe various aspects and embodiments of the invention, “synthetic image”and “synthetic data” refer to generated data or generated images, whichcan be produced by CGI. A “synthetic dataset” is a dataset that includesonly synthetic data; a synthetic dataset is used for/in model training.In accordance with the various aspects and embodiments of the invention,a “hybrid dataset” is a dataset that includes both synthetic data andreal data.

In accordance with the various aspects of the invention, trainingdatasets are generated and the system is trained using the trainingdatasets. A training dataset is developed from seed data and, inaccordance with one aspect of the invention, includes synthetic data.Synthetic data includes artificial data that replicates or mimics realworld sensory images. In accordance with an aspect of the invention, newsynthetic data can be developed using existing synthetic data to createdatasets, which may be synthetic dataset or hybrid datasets.

In accordance with the various aspects and embodiments of the invention,the synthetic data is generated using either Computer Generated Imagery(CGI), generative machine learning models, or a combination of bothapproaches. In accordance with the various aspects of the invention, thedata that makes up the dataset, which is used for training the system,is also labeled as part of the dataset generation, i.e., augmented withclass labels, object detection bounding boxes, segmentation masks, depthvalues, or other labels. Thus, supplementing or augmenting real datawith synthetic data to develop training datasets allows for scaling,diversity, and accuracy in the training datasets for application in manydifferent fields of use and application in many different areas. Inaccordance with the various aspects and embodiments of the invention,the training datasets can be used in any number of applications,including visual or face recognition, autonomous vehicles, satelliteimagery, drone imagery, gesture recognition, navigation, interiormapping, medical application and imagery, retail spaces, gazeestimation, robotics and control systems, animal agriculture,aquaculture, security systems, mobile or personal devices that performrecognition tasks (such as recognizing objects in images or landmarks onobjects), personal shopping, etc. The training datasets developedaccording to the various aspects of the invention increase performanceand efficiency of training the system.

In accordance with the various aspects of the invention, improvedtraining of machine learning models is achieved using supplementing ofreal data with the synthetic dataset. This improves training of machinelearning models for computer vision tasks, including, but not limitedto, image classification, object detection, image segmentation, andscene understanding.

Referring now to FIG. 1, in accordance with an aspect of the invention,the process 100 for generating training data is shown. At step 102, atleast a pair of seed images are identified. The seed images include thesubject that is the focus for training the model. For example, inaccordance with the various aspects and embodiments of the invention,the subject is visual recognition for facial analysis. The seed dataincludes faces. With the seed data including the subject, the seed datais used to generate the synthetic data as outline herein. In accordancewith an aspect of the invention, the synthetic data is generated usingcomputer graphics to prepare artificial imagery with objects similar tothe objects present in the seed data. In accordance with an aspect ofthe invention the synthetic data supplements neural network training byadding the synthetic data produced to the seed data to create a hybriddataset.

For example, if the system does visual recognition, then the subject maybe faces represented in the seed data: put another way, the seed datawould include images of faces. In accordance with the various aspectsand embodiments of the invention, at step 104, the seed data is used togenerate a diverse set of synthetic data representing faces. Usingcontemporary rendering techniques combined with generative deep learningmodels, a pair of seed images (seed data) are used to generate syntheticimages. In accordance with some aspects of the invention, the syntheticdata is produced from a childhood memory of the artist, which is theseed data, with no reference to real images. Thus, the scope of theinvention is not limited by the source of the seed data that the user orartist utilizes to create (or generate) the synthetic data.

In accordance with an aspect of the invention, the synthetic imagesrepresent a series of snapshots taken at different stages of a morphingprocess. This results in creating a large number of synthetic imagesthat mimic the subject.

At step 106, the system generates a training dataset. In accordance withan aspect of the invention, the system combines the created syntheticimages with real images to produce the training dataset. In thisexample, the training dataset is a hybrid dataset that includes bothreal images and synthetic images, wherein a face is the subject of theimage in accordance with the various aspects of the invention. Inaccordance with an aspect of the invention, the system generates thetraining dataset using purely synthetic image, with no real data. Inthis example, the training dataset is a synthetic dataset that is usedin the training process and no real images are used in the trainingprocess.

At step 108, the training dataset is expanded by changing variousparameters, such as camera angle, background, lighting environment,facial features, etc. As synthetic images are generated, parametersassociated with each synthetic image can be varied. There is no limit tothe number of parameters of each synthetic image that can be altered.For example, camera attributes, which are used in generating eachsynthetic image, can be varied. Camera attributes include at least imagemodality (optical, IR, multispectral, etc.), resolution, field of view,capture angle, distance from camera, ISO, white balance, exposure, hue,and saturation. Other parameters that can be altered includeenvironmental parameters. Environmental parameters include at least:lighting type (e.g. outdoor lighting, time of day, indoor lighting, LEDlighting, etc.), lighting source, lighting position, lighting intensity,background content, surface properties, etc. Other parameters that canbe identified, labeled and altered include: scene complexity based onthe number of objects in the scene, the type of objects in the scene,position of objects in the scene relative to other objects, obstructionof the subject object and other objects, static objects in the scene vsdynamic objects in the scene that change position from one moment toanother moment in time. All of the parameters are part of each datainstance that make up the training dataset and the parameters arelabeled as part of generating the synthetic data.

Thus, the generated training dataset includes a number of facial imageswithin the bounds of natural possibilities for the subject that is thefocus for training the system. The synthetic data streamlines theprocess of developing training datasets and vastly expands the size ofthe available training datasets that can be used to train the system.

Referring now to FIG. 2A, a process 200 is shown in accordance with thevarious aspects and embodiments of the invention for improving syntheticdata used for training machine learning models with generative models.The process 200 includes analyzing seed images at step 202. The seedimages are analyzed to segment the image to allow the subject to beidentified within the seed image with respect to the remainder of theseed image. In accordance with some aspects and embodiments of theinvention, the real image is processed to generate or create arasterized image. In accordance with various aspects of the invention,object recognition with an image uses segmentation and relies upontessellation of an image into superpixels. In accordance with variousaspects and embodiment of the invention, the rasterized image includessuperpixels, which represent a grouping of pixels that perceptuallybelong together and are better aligned with intensity edges than arectangular patch.

In accordance with some aspects of the invention, each superpixel in therasterized image can be altered or changed to generate a very diverserange of alterable parameters that make up synthetic images. Thesynthetic images (synthetic data) include pixel-precise labelling toallow object detection, segmentation, key feature identification andlabeling, and specific feature identification and labeling. Inaccordance with various aspects and embodiments of the invention,subject identification within the data (or image) includes preciselabeling of a range of or a set of pixels that fall within theboundaries defined by the subject of the data or image. Thus, the systemcan precisely label each pixel associated with the subject or thefeature identified.

At step 204 key features in the synthetic image are identified andlabeled. in accordance with one aspect of the invention, the labels canbe the same labels used on the seed image. There are many different keyfeatures that can be selected and labeled, a few examples of which areoutlined herein. For example and in accordance with an aspect of theinvention, key facial features that can be labelled include eyes, nose,and mouth. Each key facial feature that is labelled will include a rangeof pixels in the pixel image.

At step 206, in accordance with an aspect of the invention, superpixelsare altered in the image. In accordance with one aspect of theinvention, the image may be a real image that is rasterized. Inaccordance with another aspect of the invention, the image may be asynthetic image. The superpixels are altered to generate a diverse rangeof synthetic images based on the key features that were selected. Thesystem has a high degree of control over the superpixels and, thus, ahigh degree of control over the parameters that can be varied togenerate the synthetic images. In accordance with one aspect of theinvention, the synthetic data is generated by alterations or variationsin or to the seed data to generate synthetic images. In accordance withsome aspects of the invention, alterations can be made to existingsynthetic data to generate new synthetic data from existing syntheticdata. In accordance with one aspect of the invention, alteration aremade to superpixels. In accordance with one aspect of the invention,alteration are made to segments. As noted, labels can be used with thegenerated synthetic data.

In accordance with the various aspects of the invention, at step 208,the synthetic data is used to supplement, enhance or augment the seeddata or real data, which are images in the current example. This resultsin a training dataset. The training dataset includes a wide range ofdiversity that is quickly and easily generated. In accordance with someaspects of the invention, the training dataset is made up of real data.In accordance with one aspect of the invention, the training dataset ismade up of real data and synthetic data resulting in a hybrid dataset.The training dataset is provided to the system or model and used totrain the system or model or any AIS. For example and in accordance withsome aspects of the invention, Generative Adversarial Networks (GANs)are used to generate or enhance synthetic data.

In accordance with one aspect of the invention, once the model istrained, then the trained model can be used on new data (new images),which can include synthetic data, real data, or a combination thereof(such as a hybrid dataset). This allows the system to have a feedbackloop. The feedback loop would allow the system to analyze images usingtrained model. The analysis performed by the trained model on additionaldata would be analyzed, as noted below. The system uses the analysis togather additional information. The system can use the additionalinformation to infer or generate the appropriate labels or attributesfor the subject in the newly generated data. In accordance with someaspects of the invention, the appropriate labels or attributes used maybe selected from an existing set of labels. In accordance with someaspects of the invention, the appropriate labels or attributes used maybe new labels that are created based on analysis of the feedback, asoutlined below.

At step 212, if there is feedback from the system or the model, then atstep 214 the feedback is used to provide additional training datasets,improved training datasets, and/or update the training dataset.

In accordance with the various aspects of the invention, using thetraining data, the system can construct a synthetic dataset for facialrecognition systems. Various facial features can be generated andmodeled to further calibrate and train the system at a detailed level.For example and in accordance with the various aspects of the invention,any combination of age, gender, race, or any physical attribute ismodeled for training the system. For example, the feedback from thesystem maybe that at least two images that are difficult todifferentiate or label. While the example is provided and discussed withtwo images, any number of images may be difficult for the system todifferentiate; these images become the seed images for generating a newtraining dataset to help train the system to be able to differentiatebetween the two seed images.

The feedback is analyzed and new training datasets are generated basedon the two seed images. As new training datasets are used to furthertrain the model, the model is further improved and/or enhanced. As themodel is trained further, the model can be used on images that aredifficult to differentiate and/or label. This will result in additionalfeedback based on using the model after additional training. The newtraining dataset is used to help train the system to differentiatebetween the two seed

In accordance with one aspect of the invention, the new trainingdataset, which is used to help train for differentiation between thesimilar seed images, is generated automatically. In accordance with oneaspect of the invention, the new training dataset, which is used to helptrain for differentiation between the similar seed images, is generatedmanually by a user. In accordance with one aspect of the invention, thenew training dataset, which is used to help train for differentiationbetween these similar images, is generated remotely using a combinationof user input and auto-generation. In these various aspects of theinvention, the labeling is auto-generated. In these various aspects ofthe invention, the labeling is semi-auto generated. In these variousaspects of the invention, the labeling is manually provided by a user.

Referring now to FIG. 2B, a process 250 is shown for segmenting realimages to generate synthetic images. In accordance with the variousaspects of the invention, improved training of machine learning modelsis achieved by supplementing real data with synthetic data. Inaccordance with the various aspects of the invention, synthetic datageneration is achieved by applying image segmentation models to realdata and subsequently pasting segmentation results onto differentimages. In accordance with some aspects of the invention, the resultingimage can be automatically refined. In accordance with some aspects ofthe invention, the resulting image is not automatically refined and canbe manually refined. At step 252 images are collected or captured. Theimages include at least one object that is the subject or target of thetraining for the model. In accordance with some aspects of theinvention, real images, which include known labels, are collected orused. The real images include objects. The individual objects in thereal image are identified. In accordance with various aspects of theinvention, synthetic images with known labels can be collected andsegmented.

At step 254, the image is segmented. The images are segmented with aknown model for image segmentation; any segmentation model can be used,including classical computer vision models such as GrabCut or deeplearning based models such as Mask-RCNN or DeepMask. The segmented imageincludes at least one segment with the object that has a known label. Asa result and in accordance with some aspects of the invention, segmentedobjects with known labels, which are derived from real data, areproduced. In accordance with some aspects of the invention, segmentedobjects with known labels are derived from synthetic data. At step 256,segmented objects are pasted onto different backgrounds to produce anunrefined synthetic datasets. In accordance with the various aspects ofthe invention, the background can be synthetic. For example, produced bycomputer graphics. In accordance with the various aspects of theinvention, the background can be real or derived from a real image. Thebackground can come from unlabeled datasets because they do not have tocontain the objects in question; the objects, which are labelled, willbe pasted onto the background through a cut-and-paste process. Thesystem's use of unlabeled datasets simplifies data collection fortraining of a model and greatly expands the amount of data available forthe process. As a result of this step, a large synthetic dataset isgenerated that includes known real objects pasted onto differentbackgrounds, which provides the necessary range of diversity.

Direct cut-and-paste onto a different background, even if thesegmentation mask is perfect, may not result in a realistic image. Thus,the synthetic dataset can still be further improved for training AIS,especially when the resulting image is not realistic. Moreover,artifacts on the border of the pasted object may make it significantlyeasier to find for an AIS, and this reduces the value of the syntheticdataset.

At step 258, the system performs automatic refinement of the resultingsynthetic dataset, which can be used as the training dataset. This is atransformation that makes the images (images that result from thecut-and-paste process) more realistic while preserving the known labelsassociated with the object. The refinement is outlined with respect toFIG. 4, wherein a hybrid neural network serves as a refiner to enhancethe training dataset. As noted above with respect to the various aspectsof the invention, the training dataset may be a real training dataset,synthetic training dataset, or a hybrid training dataset that is made upof a combination of real and synthetic data.

In accordance with some aspects and embodiments of the invention, therefiner uses conditional generative models trained to make an imageindistinguishable from a real image, as outline with respect to therefiner 502 of FIG. 5. In accordance with some aspects and embodimentsof the invention, training such models requires only an unlabeleddataset of real images. Additionally, this dataset does not have tocontain the exact object that the model of the system is intended toultimately recognize. This greatly simplifies data collection needed fortraining the model.

Referring now to FIG. 3, various 3D model generations of a face areshown, such as a face 300, a face 302, and a face 304. Each face shows adifferent facial feature used to generate the synthetic images for thetraining data. Through variations of the various parameters, additionaltraining datasets are generated. Facial features in the face 300, forexample, includes a brow ridge that is different from facial features inthe face 302 and facial features in the face 304. Many other facialfeatures of the faces can be varied, including eye size (large tosmall), width of chin (wide to thin), face (heavy to thin), nose (flatto pointed) (narrow to wide) (down to up), shape of mouth and lips (fullto slim), etc. The face 300, the face 302, and the face 304 are shownusing similar camera or vision angles.

In accordance with the various aspects and embodiments of the invention,the camera angle can be varied so that a face is presented or being seenfrom various angles or viewpoints to represent a 3-dimensional image.The system can generate additional synthetic data through variations ofthe camera angle or viewpoint angle.

Referring now to FIG. 4, a system 400 is shown for generating trainingdatasets, which may include hybrid datasets with real data and syntheticdata. The system 400 includes a simulation platform 402 that generatessynthetic images 404. In accordance with one embodiment of theinvention, the synthetic images 404 are combined with real images 406 bya hybrid neural network 408. In accordance with one embodiment of theinvention, the training dataset is limited to synthetic images 404 thatare provided to the network 408.

The network 408 generates enhanced training data 410 to provide atraining dataset. The training dataset is used for further training thesystem 400 using the neural network training module 412 to produce anenhanced model 414.

Referring now to FIG. 5, the network 408 includes a Refiner® 502 thatreceives the synthetic images 404. The refiner 502 is in communicationwith a discriminator (D) 506 and a comparator 514. Synthetic images 404and enhanced training data 410 are provided as inputs to the comparator514. The discriminator 506 and the comparator 514 provide inputs to therefiner 502. The refiner 502 generates the enhanced training data 410that includes domain matched (DM) images. In accordance with oneembodiment of the invention, the discriminator 506 receives the realimages 406 and the enhanced training data 410 to provide an input to therefiner 502. The discriminator 506 also generates, as an output 508,information about of real images and DM images.

Referring now to FIG. 6A, a system 650 is shown, in accordance withvarious aspects of the invention, that uses an adaptive process thatchanges based on the properties of obtained synthetic data. The system650 includes: an AIS (computer vision model) 652 that is trained on asynthetic dataset, a synthetic data generator (SDG) 654 that has tunableparameters or some other way to change the generation process, and amodel analyzer unit (MAU) 656 designed to process the final and/orintermediate outputs of the AIS 652 and establish what changes should bedone to the parameter values or otherwise in the generation process inthe SDG 654 in order to improve the quality (accuracy or otherevaluation metric) of the AIS 652. The AIS 652. the SDG 654, and the MAU656 are in communication using an automated feedback loop. In accordancewith one aspect and embodiment of the invention, the SDG 654 begins togenerate synthetic data with some default set of parameters/options. Theresulting data is fed to the AIS 652. The system being training of theAIS 652. During training, the MAU 656 has access to the outputs of AIS652 and intermediate outputs depending on the structure of the AIS 652,for example, the outputs of intermediate layers in a deep neuralnetwork. Based on this information and the quality metrics for AIS 652measured on synthetic and/or real validation sets, the MAU 656automatically determines what would constitute a favorable change in theparameter values or generation options for the SDG 654. The MAU 656provides the information to the SDG 654. Then the values are changed andthe SDG 654 begins generating a slightly different stream of syntheticdata, thus closing the feedback loop. Then the process is repeated,which iteratively refines synthetic data, in order to improve theevaluation metrics produced by the AIS 652.

Referring now to FIG. 6B, a process 600 for receiving feedback is shownin accordance with the various aspects of the invention. At step 602 thesystem generates synthetic images (data) that mimic real images (data),as outline above. In accordance with one aspect of the invention,synthetic data can be enhanced with generative machine learning models.In accordance with an aspect of the invention, the synthetic data isprepared using computer graphics. In accordance with one aspect of theinvention, the synthetic data is used to train a generative machinelearning model, which in turn is used to make synthetic data (or images)more realistic. At step 604, the synthetic data is combined with realdata. In accordance with an aspect of the invention, the system cantrain a model for visual recognition by using the synthetic data, whichis enhanced by the generative model, with the training dataset. At step606 the training dataset if generated. At step 608 the training datasetis used to train the model. The precision of the trained model can betested on new data. In accordance with one aspect of the invention, theprecision of the trained model is tested using new real data. Inaccordance with one aspect of the invention, the precision of thetrained model is tested using new synthetic data. The testing process ofthe trained model can produce feedback, which the system can use tofurther refine and train the model. At step 610, the model providesfeedback that is used to enhance, update or improve the trainingdataset, which is used to refine or further train and improve the model.

Referring now to FIG. 7, a process 700 for combining real images togenerate synthetic images is shown. The process 700 includes capturingreal images at step 702. At step 704, the real images are segmented. Thesegmentation separates the different objects in the real image. At step706 segments from different real images are combined to generate newsynthetic images. Any number of new synthetic images can be generatedbased on the variation of combinations of the various segments of thereal images. In accordance with one aspect of the invention, at step 708a training dataset is generated using the generated synthetic images. Inaccordance with one aspect of the invention, the training datasetincludes real images (captured and labelled) and synthetic images(generated and labelled). In accordance with one aspect of theinvention, the training dataset includes only real images, wherein thereal images are captured and labelled. In accordance with one aspect ofthe invention, the training dataset includes only synthetic images,which are labelled as outlined above. At step 710 the training datasetis used to train the model.

In accordance with various aspects and embodiments of the invention, thereal image, which is segmented, is used to generate synthetic data byremoving objects or portions of the real image. In this way, the absenceof an object in the synthetic data can be used to further train thesystem or model. The synthetic data with a missing object is presentedto the system. The response expected from the model would be anindication that the object, which is the subject to be detected by themodel, is not present or missing. For example, a nose that is removedfrom a segmented image, which still includes other facial features, canbe used to train the model to detect the absence or lack of a nose inthe image. In this example and in accordance with one aspect of theinvention, the image presented to the model may be an altered real imagethat is modified by removing the object (nose). In this example and inaccordance with another aspect of the invention, the image presented tothe model may be a synthetic data that is generated based on the realimage.

In accordance with the various aspects of the invention, syntheticdatasets are implemented for domain adaptation and domain transfertechniques for training of machine learning models. Domain adaptation isthe problem of leveraging labeled data in a source domain to learn andtrain an accurate model in a target domain, wherein the labels arescarce or unavailable. In regard to using synthetic datasets, domainadaptation is applied to a machine learning model trained on one datadistribution, which is a source domain (in this case, the domain ofsynthetic data), so that the model solves similar problems on a datasetof different nature, which is a target domain (in this case, the domainof real data). In accordance with one aspect of the invention,unsupervised domain adaptation is used when labeled data is available inthe source domain and not in the target domain; the target domain hasonly unlabeled data available. In accordance with one aspect of theinvention, the system applies unsupervised domain adaptation insituations where the source domain is the domain of synthetic data,which has abundant and diverse range of labeled data, and the targetdomain is the domain of real data, which includes a large dataset thatmay be unlabeled.

In accordance with aspects and embodiment of the invention, the systemuses a generalized feature extractor model, where features of the sourcedomain and target domain are made to be indistinguishable. Example ofsuch models include: RevGrad model, Virtual Adversarial DomainAdaptation (VADA) model, and Decision-boundary Iterative RefinementTraining with a Teacher (DIRT-T) model. In accordance with an aspect ofthe invention, training of the model is progressive. As the trainingprogresses, features emerge that are discriminative for the sourcedomain and indiscriminate with respect to the shift between the sourceand target domains. For example, the various aspects and embodiment ofthe invention includes a process that uses modification of domainadaptation VADA model and DIRT-T model by adding the source entropymaximization term to their respective loss functions. The system, usingthe synthetic dataset with the labels, learns a direct source-to-targettransformation, such as an image-to-image translation, using generativemodels and domain adaptation frameworks based on generative models, suchas SimGAN, CycleGAN, and Cycle-Consistent Adversarial Domain Adaptation(CyCADA). In accordance with aspects of the invention, the systemprovides an approach for unsupervised domain adaptation to thesynthetic-to-real adaptation.

In accordance with the various aspects of the invention, a computer anda computing device are articles of manufacture. Other examples of anarticle of manufacture include: an electronic component residing on amother board, a server, a mainframe computer, or other special purposecomputer each having one or more processors (e.g., a Central ProcessingUnit, a Graphical Processing Unit, or a microprocessor) that isconfigured to execute a computer readable program code (e.g., analgorithm, hardware, firmware, and/or software) to receive data,transmit data, store data, or perform methods. The article ofmanufacture (e.g., computer or computing device) includes anon-transitory computer readable medium or storage that may include aseries of instructions, such as computer readable program steps or codeencoded therein. In certain aspects of the invention, the non-transitorycomputer readable medium includes one or more data repositories ormemory. Thus, in certain embodiments that are in accordance with anyaspect of the invention, computer readable program code (or code) isencoded in a non-transitory computer readable medium of the computingdevice. As outlined herein, a processor or a module, in turn, executesthe computer readable program code to create or amend an existingcomputer-aided design using a tool. The term “module” as used herein mayrefer to one or more circuits, components, registers, processors,software subroutines, or any combination thereof. In accordance withsome aspects and embodiments of the invention, the creation or amendmentof the computer-aided design is implemented as a web-based softwareapplication in which portions of the data related to the computer-aideddesign or the tool or the computer readable program code are received ortransmitted to a computing device of a host.

An article of manufacture or system, in accordance with various aspectsof the invention, is implemented in a variety of ways: with one or moredistinct processors or microprocessors, volatile and/or non-volatilememory and peripherals or peripheral controllers; with an integratedmicrocontroller, which has a processor, local volatile and non-volatilememory, peripherals and input/output pins; discrete logic whichimplements a fixed version of the article of manufacture or system; andprogrammable logic which implements a version of the article ofmanufacture or system which can be reprogrammed either through a localor remote interface. Such logic could implement a control system eitherin logic or via a set of commands executed by a processor.

Accordingly, the preceding merely illustrates the various aspects andprinciples as incorporated in various embodiments of the invention. Itwill be appreciated that those of ordinary skill in the art will be ableto devise various arrangements that, although not explicitly describedor shown herein, embody the principles of the invention and are includedwithin its spirit and scope. Furthermore, all examples and conditionallanguage recited herein are principally intended to aid the reader inunderstanding the principles of the invention and the conceptscontributed by the inventors to furthering the art and are to beconstrued as being without limitation to such specifically recitedexamples and conditions. Moreover, all statements herein recitingprinciples, aspects, and embodiments of the invention, as well asspecific examples thereof, are intended to encompass both structural andfunctional equivalents thereof. Additionally, it is intended that suchequivalents include both currently known equivalents and equivalentsdeveloped in the future, i.e., any elements developed that perform thesame function, regardless of structure.

Reference throughout this specification to “one embodiment,” “anembodiment,” or “in accordance with some aspects” and similar languagemeans that a particular feature, structure, or characteristic describedin connection with the various aspects and embodiments are included inat least one embodiment of the invention. Thus, appearances of thephrases “in accordance with an aspect,” “in accordance with oneembodiment,” “in an embodiment,” “in certain embodiments,” and similarlanguage throughout this specification refer to the various aspects andembodiments of the invention. It is noted that, as used in thisdescription, the singular forms “a,” “an” and “the” include pluralreferents, unless the context clearly dictates otherwise.

The described features, structures, or characteristics of the inventionmay be combined in any suitable manner in accordance with the aspectsand one or more embodiments of the invention. In the followingdescription, numerous specific details are recited to provide anunderstanding of various embodiments of the invention. One skilled inthe relevant art will recognize, however, that the invention may bepracticed without one or more of the specific details, or with othermethods, components, materials, and so forth. In other instances,well-known structures, materials, or operations are not shown ordescribed in detail to avoid obscuring the aspects of the invention.

Additionally, it is intended that such equivalents include bothcurrently known equivalents and equivalents developed in the future,i.e., any elements developed that perform the same function, regardlessof structure. The scope of the invention, therefore, is not intended tobe limited to the exemplary embodiments shown and described herein.

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. The verb couple, its gerundialforms, and other variants, should be understood to refer to eitherdirect connections or operative manners of interaction between elementsof the invention through one or more intermediating elements, whether ornot any such intermediating element is recited. Any methods andmaterials similar or equivalent to those described herein can also beused in the practice of the invention. Representative illustrativemethods and materials are also described.

All publications and patents cited in this specification are hereinincorporated by reference as if each individual publication or patentwere specifically and individually indicated to be incorporated byreference and are incorporated herein by reference to disclose anddescribe the methods and/or system in connection with which thepublications are cited. The citation of any publication is for itsdisclosure prior to the filing date and should not be construed as anadmission that the invention is not entitled to antedate suchpublication by virtue of prior invention. Further, the dates ofpublication provided may be different from the actual publication dateswhich may need to be independently confirmed.

Therefore, the scope of the invention is not intended to be limited tothe various aspects and embodiments discussed and described herein.Rather, the scope and spirit of the invention is embodied by theappended claims.

What is claimed is:
 1. A method for using feedback in an adaptivegeneration system, the method comprising: generating a training datasetincluding a plurality of images; providing the training dataset to agenerative model to train the generative model; receiving feedback fromthe generative model identifying a first image and a second image thatrequire further labeling for differentiation; generating a new trainingdataset targeted at training the system to differentiate between thefirst image and the second image; adapting the generative model trainedusing the new training dataset; and updating the training dataset withthe new training dataset to generate an enhanced dataset.
 2. The methodof claim 1, wherein the system includes a machine learning (ML) model.3. The method of claim 1, wherein the system includes deep learning (DL)model.
 4. The method of claim 1, wherein the system is an artificialintelligence system (AIS).
 5. The method of claim 1, wherein the step ofgenerating includes procedural generation.
 6. The method of claim 1,wherein the step of generating includes computer graphic techniques. 7.The method of claim 1, wherein the plurality of images are real images.8. The method of claim 1, wherein the plurality of images are syntheticimages.
 9. The method of claim 1, wherein the plurality of images are acombination of real images and synthetic images.
 10. The method of claim9 wherein the synthetic images statistically replicate the real images.11. The method of claim 1, wherein the trained model is used to performvisual recognition.
 12. The method of claim 11, wherein the visualrecognition includes at least one of image classification, objectdetection, object segmentation, and scene understanding.
 13. The methodof claim 1 further comprising the step of pretraining the generativemodel using real data.
 14. The method of claim 1 further comprising thestep of pretraining the generative model using synthetic data.
 15. Themethod of claim 1 further comprising the step of pretraining thegenerative model using real data and synthetic data.
 16. The method ofclaim 1, wherein the step of generating includes using real data basedon real images with at least one image that is generated using agenerative adversarial network (GAN) and at least one image that isgenerated using computer generated imagery (CGI).
 17. The method ofclaim 1, wherein the step of generating the new training datasetincludes changing at least on parameter of at least one of the firstimage and the second image to generate a new synthetic data.